ABSTRACT
OBJECTIVE: To develop a machine learning framework to forecast emergency department (ED) crowding and to evaluate model performance under spatial and temporal data drift. MATERIALS AND METHODS: We obtained four datasets, identified by the location: 1-large academic hospital and 2-rural hospital, and time period: pre-COVID (Jan 1, 2019-Feb 1, 2020) and COVID-era (May 15, 2020-Feb 1, 2021). Our primary target was a binary outcome that is equal to 1 if the number of patients with acute respiratory illness that were ED boarding for more than four hours was above a prescribed historical percentile. We trained a random forest and used the area under the curve (AUC) to evaluate out-of-sample performance for two experiments: 1) we evaluated the impact of sudden temporal drift by training models using pre-COVID data and testing them during the COVID-era, 2) we evaluated the impact of spatial drift by testing models trained at Location 1 on data from Location 2, and vice versa. RESULTS: The baseline AUC values for ED boarding ranged from 0.54 (pre-COVID at Location 2) to 0.81 (COVID-era at Location 1). Models trained with pre-COVID data performed similarly to COVID-era models (0.82 vs. 0.78 at Location 1). Models that were transferred from Location 2 to Location 1 performed worse than models trained at Location 1 (0.51 vs. 0.78). DISCUSSION AND CONCLUSION: Our results demonstrate that ED boarding is a predictable metric for ED crowding, models were not significantly impacted by temporal data drift, and any attempts at implementation must consider spatial data drift.